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ABSTRACT 

The  term  virtual  reality  can  encompass  varying  meanings,  but  some  generally  accepted 
attributes  of  a  virtual  environment  are  that  it  is  immersive,  interactive,  and  intuitive. 
Integration  of  many  different  technologies  is  necessary  in  order  to  produce  a  system 
having  these  attributes.  Applications  such  as  molecular  modeling,  flight  simulators, 
physical  rehabilitation,  and  architectural  design  and  walk  through  utilize  various  pieces  of 
the  technology,  but  at  present  there  are  few  practical  applications  which  are  utilizing  the 
broad  range  of  virtual  reality  technology. 

This  paper  will  discuss  an  application  to  provide  a  sonar  operator  with  a  virtual  underwater 
acoustic  environment  including  the  design  methodology  and  human  factors  experiments 
for  testing  the  system.  The  technologies  used  in  this  application  are  3-D  modeling  software, 
computer  image  generation  software,  3-D  graphics  rendering  engines,  helmet  mounted 
stereoscopic  displays,  head  tracking  sensors,  3-D  input  devices,  voice  recognition,  and  a  3-D 
spatializing  sound  system. 

INTRODUCTION 

There  are  many  different  opinions  as  to 
what  defines  virtual  reality  (VR)  and  what 
elements  are  essential  in  creating  a 
virtual  environment,  but  some  generally 
accepted  attributes  of  the  environment 
are  that  it  is  immersive,  interactive,  and 
intuitive.  Some  of  the  basic  elements 
needed  to  create  a  virtual  environment 
are  computer  generated  imagery,  3-D 
input  devices,  position  trackers,  and  a 
helmet  mounted  or  monitor  based  display. 

New  technologies  are  emerging  and 
evolving  almost  daily  that  further  the 
capabilities  of  VR  developers  to  create 
these  environments.  This  paper  will 
briefly  introduce  some  of  these 
technologies  before  describing  the 
Advanced  Technology  ASW  Displays 
(ATAD)  project,  its  design  methodology 
and  system  evaluation. 


Enabling  Technologies 
Computing  Power 

Computing  power  is  the  backbone  of  any 
VR  project,  and  now  specialized  graphics 
computers  are  able  to  produce  startlingly 
realistic  computer  generated  images  with 
the  aid  of  lighting  models  and  texture 
mapping.  The  Silicon  Graphics  Reality 
Engine  product  line  and  the  Evans  & 
Sutherland  ES1G  2000  series  computers  can 
generate  texture  mapped  polygons  in 
hardware  at  update  rates  more  than 
adequate  to  accommodate  real  time 
movement  through  an  environment.  The 
incorporation  of  reduced  instruction  set 
components  (RISC),  application  specific 
integrated  circuits  (ASICs)  and  multiple 
processors  makes  the  current  generation 
of  graphics  computers  capable  of 
performance  previously  the  domain  of  the 
supercomputer  at  a  fraction  of  the  cost, 
liven  lower  end  graphics  computers  are 
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capable  of  generating  high  qualm 
images,  with  the  performance  tradeoff 
being  the  complexity  of  the  scene  versus 
the  speed  of  the  update  rate. 

Software 

The  software  tools  available  for  graphics 
computers  vary  in  capability  and  play  an 
important  part  in  how  quickly  and  easily 
one  can  develop  a  virtual  environment. 
The  different  objects  that  appear  in  a 
scene  can  be  generated  by  programming 
with  a  graphics  library  such  as  the  Silicon 
Graphics  GL  or  a  similar  graphics  library 
on  another  platform.  This  is  a  tedious 
process  that  involves  figuring  out  the 
geometry  of  each  polygon  in  x,  y,  and  z 
coordinates.  This  can  be  especially 
complicated  considering  that  a  reasonably 
realistic  model  of  an  object  such  as  a 
submarine  can  have  a  thousand  polygons 
or  more.  3-D  modeling  software  products 
now  exist  which  allow  the  user  to  directly 
draw  an  object  in  three  dimensions  on 
screen  and  apply  characteristics  to  the 
object  such  as  material  properties,  levels 
of  detail  and  articulation  points,  and 
interactively  see  the  results  of  changing 
various  properties.  Other  software  tools 
are  available  to  design  user  interfaces, 
serve  as  simulation  managers,  import  CAD 
and  other  3-D  model  formats  and  address 
other  aspects  of  building  a  virtual 
environment. 

Position  Trackers 

A  virtual  environment  needs  position 
trackers  and  input  devices  in  order  to 
make  it  interactive.  Two  of  the  most 
widely  used  position  trackers  use  an 
electromagnetic  field  source  and  sensor, 
each  connected  to  a  control  unit,  to  detect 
six  degrees  of  freedom:  position  in  x,  y, 
and  z  from  the  source,  and  pitch,  roll,  and 
yaw  rotations.  Mechanical  linkages  and 
ultrasonic  devices  are  methods  that  some 
other  position  trackers  use.  A  position 
tracker  enables  the  graphics  to  change 
with  relation  to  the  user's  head  position  so 
the  environment  behaves  just  as  it  would 
if  the  person  were  looking  at  a  scene  in 
real  life.  The  ability  to  passively  track 
where  the  person  wants  to  look  instead  of 
controlling  the  scene  through  a  keyboard 
or  mouse  adds  greatly  to  the 


lmimrsiveness  and  the  intunivencss  of 
the  environment,  since  the  user  simply 
does  what  comes  naturally  to  view  a 
dillerent  par  t  <>l  the  scene. 

Input  Devices 

In  order  for  the  user  to  have  any 
interaction  with  the  environment,  he  or 
site  needs  input  devices,  and  there  are 
several  types  available.  At  least  two  input 
dev  ices  are  based  on  gloves  that  have  an 
attachment  for  an  electromagnetic 
position  tracker  and  use  thin  fiber  optic 
cables  running  along  the  fingers  to  detect 
joint  bending.  The  user  signifies 
different  commands  to  perform  by  wav  of 
a  set  of  gestures  that  is  developed  for  the 
application.  There  is  also  at  least  one 
product  that  is  an  exoskeleton  that  fits 
over  the  hand  and  uses  robotics 
technology  to  measure  hand  and  finger 
motion.  Yet  another  device  is  designed  as 
a  2-D  or  3-D  mouse.  The  device  functions 
as  a  regular  optical  mouse  when  moved  on 
the  mouse  pad,  and  has  an  electromagnetic 
sensor  inside  which  enables  it  to  track 
position  when  raised  off  the  pad.  The  user 
can  then  use  the  three  buttons  for 
different  commands  in  the  3-D 
environment.  None  of  these  d-vDes  is 
entirely  intuitive  and  the  software 
interfaces  are  primitive,  but  an  easy  to  use 
input  device  for  VR  is  on  the  horizon. 

Head  Mounted  Displays 

An  essential  part  of  any  virtual 
environment  is  the  display.  While  a  head 
mounted  display  (HMD)  is  not  essential  for 
VR,  it  is  the  one  device  that  is  most  closely- 
associated  with  virtual  reality.  Several 
options  arc  available  in  HMDs,  generally 
determined  by  four  factors:  color  or 
monochrome,  cathode  ray  tube  (CRT) 
based  or  liquid  crystal  display  (LCD)  based. 
Small,  very  high  resolution  CRTs  are 
widely  available  in  monochrome,  but  only- 
very  recently  have  color  versions  become 
available.  Some  disadvantages  of  CRTs  are 
ih'Hr  weight  and  high  power 
consumption.  Small  color  LCDs  that  don't 
have  these  disadvantages  are  available, 
but  at  present  the  resolution  of  an  I  CD 
display  is  much  less  than  what  is  available 
in  CRTs.  Advances  in  both  of  these 
technologies  are  progressing  to  the  point 
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where  it  will  only  be  a  matter  of 
preference  as  to  which  type  of  HMD  to  use. 

3-D  Sound 

Another  technology  that  greatly  adds  to 
the  immersiveness  of  the  environment  is 
3-D  sound  technologs.  A  device  called  the 
Convolvotron  is  capable  of  taking  up  to 
four  real  time  analog  input  sounds  and 
spatializing  them  through  stereo 
headphones  to  sound  like  they  are  coming 
from  separate,  external  points  in  the 
environment.  When  combined  with  a 
position  tracker,  the  user  hears 
apparently  stationary  sounds  as  he  or  she 
moves  around  them.  The  addition  of  sound 
to  the  environment  adds  greatly  to  the 
feeling  of  immersion,  and  is  essential  to 
the  virtual  underwater  acoustic 
environment  of  the  ATAD  project. 

ATAD  Design  Methodology 
Design  Objective 

The  Advanced  Technology  ASW  Displays 
project  is  funded  through  the  Human 
Factors  block  of  the  Manpower,  Personnel 
&  Training  Office  of  the  Chief  of  Naval 
Operations.  Its  overall  mission  is  to 
increase  the  effectiveness  of  anti¬ 
submarine  warfare  (ASW)  operators  on  all 
platforms  (afloat,  airborne,  and  ashore). 
The  first  priority  is  to  improve  shipboard 
sonar  operators'  performance.  The  idea  is 
that  performance  can  improve  without 
changing  the  actual  sonar  sensors,  but  by 
simply  modifying  the  interface  to  make 
the  system  easier  for  the  operator  to  use. 
A  3-D  system  could  present  the  critical 
tactical  information  in  a  far  more 
intuitive  and  integrated  wav  than  the 
existing  2-D  displa\s,  therefore  reducing 
the  cognitive  interpretation  burden  and 
learning  time  while  improving  the 
tactical  success  rate.  The  ATAD  virtual 
environment  approach  aims  to  provide 
the  operator  with  a  physical 
representation  of  the  ocean  environment 
with  overlaid  representations  of 
information  such  as  latitude  and  longitude 
lines,  '^ertnal  layers,  and  the  sonar  data 
from  shipboard  sensors  (Fig.  1). 

Designing  a  virtual  environment  from  the 
ground  up  encompasses  quite  a  few 
challenges.  Deciding  which  hardware 


will  suit  the  design  and  budget  constraints 
requires  continual  research  due  to  the 
constanth  changing  market.  A  difficult) 
in  working  with  sue  h  new  technolog)  is 
integrating  the  matt)  different 
components  into  a  cohesive  system.  With 
onlv  a  small  knowledge  pool  to  draw  from, 
much  of  what  is  learned  is  determined 
from  trial  and  error. 

Hardware  Components  and  integration 
I  he  hardware  components  of  the  A  TAD 
system  are  two  graphics  computers,  a  3-D 
audio  system,  a  data  server  and  audio 
processing  computer,  position  tracker, 
head  mounted  display,  speech  recognition 
system  and  3-D  input  devices  (Fig  2). 

Two  Silicon  Graphics  4D/3J0VGX  graphics 
computers  are  the  graphics  client  hosts, 
each  one  dedicated  to  providing  the 
graphics  to  one  eye  of  the  helmet  mounted 
display.  The  ATAD  project  uses  SGI  systems 
because  of  their  wide  acceptance  in  the 
simulator  arena,  their  price  and 
reputation  as  being  among  the  leaders  in 
the  3-D  graphics  market.  In  addition 
other  branches  at  NCCOSC  RDT&F  Division 
(NRaD)  have  some  experience  with  SGI 
systems.  Fach  machine  has  one  MIPS 
R3000  RISC  central  processing  unit  (CPU) 
running  at  33  MHz  w'ith  a  capability  of  5.1 
MFL.OPS  or  million  floating  point 
operations  per  second.  Since  these 
systems  were  purchased  in  March  of  1992 
SGI  has  made  significant  improvements  in 
processor  speed  and  texture  mapping 
capability. 

The  Convolvotron  3-D  spatializing  audio 
system  from  Crystal  River  Engineering 
runs  on  a  38b  PC  with  two  Spectrum  boards 
using  the  Texas  Instruments  TMS320C25 
DSP  chip.  The  Convolvotron  makes  use  of 
research  involving  head  related  transfer 
functions  (HRTFs).  To  determine  an  HRTI , 
miniature  microphones  take 
measurements  from  within  a  subject's  ear 
canals  of  w'hite  noise  bursts  at  different 
positions  in  space.  The  Convolvotron  takes 
real-'ime  sound  input  and  convolves  ii 
with  the  HRTF  to  make  the  sound  appear  as 
if  it  is  coming  from  a  particular  location, 
and  sends  the  output  to  stereo  headphones. 
The  helmet  mounted  displav  incorporates 
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the  stereo  headphones  that  receive  the 
output.  Other  3-D  sound  systems  exist,  but 
to  date  the  Convolvotron  is  the  only  one  of 
its  quality  that  does  not  rely  on  multiple 
speakers  to  achieve  the  spatialization.  The 
use  of  headphones  makes  it  particularly 
suited  for  use  in  a  virtual  environment 
where  the  helmet  mounted  display 
encloses  the  user.  The  use  of  3-D  sound 
greatly  enhances  the  feeling  of 
immersion  in  the  environment  and  is 
essential  to  the  ATAD  project.  The 
objective  is  for  the  sonar  operator  to  be 
able  to  selectively  listen  to  multiple  beams 
of  spatially  accurate  incoming  sonar 
information  and  intuitively  know  where 
the  sounds  are  coming  from.  The 
Convolvotron  includes  a  Polhemus 
Isotrack  electromagnetic  position  tracker 
so  the  sounds  can  appear  stationary  as  the 
user  moves  his  or  her  head. 

An  SGI  Indigo  24XS  is  the  data  server  host. 
The  Indigo  runs  a  simulator  base  that 
controls  the  motion  of  the  objects  such  as 
surface  ships  and  submarines  in  the 
environment.  The  Indigo  sends  the 
simulator  update  information  over 
Ethernet  to  each  of  the  graphics  clients  . 
An  RS-232  connection  to  the  Convolvotron 
enables  the  Indigo  to  tell  the  Convolvotron 
where  to  place  the  objects  that  are  making 
sound  and  reads  the  position  data  from  the 
Isotrak.  The  Indigo  sends  the  head 
position  over  Ethernet  to  the  left  and  right 
eye  clients  to  reflect  the  head  movements 
in  the  graphics.  The  analog  audio  output 
capabilities  of  the  Indigo  will  play 
different  audio  files  from  disk  as  the  audio 
input  to  the  Convolvotron;  currently  the 
audio  input  is  a  four  track  cassette  that  is 
only  useful  if  there  are  just  four 
stationary  objects  of  interest  in  the 
environment. 

The  head  mounted  display  is  a  CRT  based 
system  that  uses  the  mounting  frame, 
CRTs,  and  electronics  from  a  teleoperated 
vehicle  program  developed  at  NRaP.  The 
CRT  mounting  frame  is  the  same  as  is  used 
for  night  vision  helmet  systems.  The  CRTs 
are  monochrome  green  NTSC  video 
resolution  that  is  the  same  as  a  standard 
television.  1'he  CRT  frame  mounts  onto  a 
bicycle  helmet  with  space  cut  out  to  make 


room  lor  the  headphones.  A 

t  ounterbalaiu  e  in  the  back  of  the  helmet 
compensates  lor  the  weight  of  the  Ckls. 
Ihe  helmet  receives  two  video  channels, 
one  from  each  graphics  client,  converted 
to  N  I  'SC  resolution  by  an  add  on  broadcast 
video  option  board.  Even  though  the 
display  is  monochrome,  and  about  one 
quarter  the  resolution  of  the  full  screen 
capability  of  the  SGI  19  inch  display  ,  the 
results  are  of  a  fairly  good  quality  . 

Ihe  ATAl)  project  has  acquired  two 
different  3-D  input  devices  to  evaluate, 
one  of  which  will  be  integrated  into  the 
system.  The  DataGlove  by  VRE  is  one  of  the 
devices,  which  uses  fiber  optic  technology 
to  measure  joint  bending  and  an  Isotrak 
tracker  attached  at  the  wrist  to  measure 
position.  The  other  device  is  the  Flying 
Mouse  by  SimGraphics  that  functions  as  a 
2-D  mouse  on  the  pad  and  has  an  Isotrak 
inside  it  to  function  as  a  3-D  controller 
when  raised  off  the  pad.  In  addition  a 
voice  recognition  system  will  work 
together  with  the  input  devices.  To  date 
very  little  has  been  done  with  3-D  input 
devices  beyond  simple  demo  programs  and 
there  are  no  standard  methods  of 
maneuvering  in  a  3-D  environment  the 
way  there  are  with  a  2-D  mouse  on  a 
screen.  Human  factors  research  is 
continuing  in  this  area  to  determine  the 
best  ways  for  people  to  interact  with  a 
virtual  3-D  environment  and  software 
tools  are  being  developed  to  enable  easier 
addition  of  the  user  interfaces  to  3-D 
applications. 

Software  Components  fc  Integration 
The  software  components  of  the 
ATAD  project  consist  of  a  3-D  modeling 
application,  a  submarine  environment 
simulator,  a  high-level  interface  for 
controlling  the  graphics,  and  system 
integration  code. 

MuItiGen  by  Software  Systems  is  the  3-D 
modeling  software  package  used  to 
generate  all  of  the  objects  that  appear  in 
the  scene  such  as  submarine  models, 
marine  animals,  ocean  bottom  terrain  and 
representations  of  the  incoming  sonar 
data.  Il  is  also  used  to  create 
representations  of  such  relevant 
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information  as  thermal  layers,  sonar 
coverage  /ones  and  latitude  and  longitude 
lines  that  would  be  of  use  to  the  sonar 
operator.  By  modeling  normally  invisible 
information  as  something  visible  in  the 
virtual  environment,  the  operator  can 
absorb  more  information  at  an  intuitive 
level.  This  frees  the  operator  to  pay 
attention  to  the  important  tasks  of 
detection  and  identification. 

A  submarine  environment  simulator 
developed  in-house  for  a  different  project 
serves  as  the  foundation  for  the  virtual 
environment.  The  simulator  provides 
realistic  models  of  submarine  movement 
and  sound  propagation  in  water.  The 
submarines,  surface  ships  and  biologies 
have  paths  that  are  scripted  in  time  to 
follow  through  the  environment.  Thus 
subjects  can  be  tested  in  different 
scenarios  without  undue  reprogramming 
of  the  simulator  by  simply  changing  the 
scripts.  The  simulator  is  written  in  C  and 
runs  on  the  SGI  Indigo  data  server  host. 

The  Gemini  Visual  System  package  (GVS) 
by  Gemini  Technology  Corporation  is  a 
library  collection  of  high-level  calls  to 
create  dynamic  simulations  using  3-D 
models.  The  library  calls  are  invoked  to 
assign  specific  3-D  models  created  in 
MultiGen  to  the  objects  described  in  the 
submarine  simulator  and  draw  the 
graphics  to  reflect  the  correct  motion  of 
all  the  objects  from  a  specified  eye  point  . 
Other  capabilities  exist  such  as  collision 
detection  between  objects,  the  ability  to 
switch  eye  points  to  a  different 
perspective,  or  animate  articulated 
portions  of  a  3-D  model. 

Tying  all  of  these  pieces  together  is  the 
system  integration  code  that  is  generated 
in-house.  The  Indigo  data  server  runs  a 
process  that  communicates  through  RS- 
232  with  the  Convolvotron  to  send  position 
data  for  up  to  four  objects  in  the  scene 
that  have  sounds  associated  with  them,  and 
receive  head  position  data  from  the 
Convolvotron's  Isotrak  tracker  mounted 
on  the  helmet.  The  Indigo  then  sends  the 
head  location  information  using  UNIX 
sockets  over  Ethernet  to  the  graphics 
servers  lor  the  left  and  right  eye  views. 


Evaluation  of  the  System 
In  order  to  evaluate  the  effectiveness  ol  a 
virtual  environment  system,  human 
factors  experiments  need  to  quantitatively 
measure  performance.  One  aspect  of  the 
sonar  problem  has  been  chosen  to  focus 
on,  namely  detection  of  a  target  in  an 
active  sonar  situation.  ATAD  team 
members  have  met  with  personnel  at  the 
Nasal  ASW  school  in  San  Diego  to  get  a 
good  idea  of  how  sonar  operators  handle 
the  problem  currently.  A  model  of  the 
existing  displays  will  be  built  using  the 
Virtual  Prototyping  System  (VAPS)  on  an 
existing  Silicon  Graphics  machine.  An 
experiment  is  being  designed  which  will 
present  multiple  targets  in  the 
environment  and  test  the  subjects  on  both 
systems  using  the  same  data  sets.  This 
experiment  will  help  to  determine  il  the 
virtual  environment  approach  does  indeed 
show  a  performance  improvement  for  at 
least  one  segment  of  the  ASW  problem.  If 
the  results  are  favorable  then  the  system 
will  be  expanded  to  encompass  more 
aspects  of  the  sonar  environment. 

As  the  technology  advances  and  interest 
in  virtual  environments  expands,  a  fair 
amount  ,  of  research  and  experimentation 
is  being  done  on  different  aspects  of 
virtual  reality  systems  such  as 
stereoscopic  displays,  3-D  spatialized  sound 
and  3-D  input  devices.  To  date  few 
experiments  of  this  kind  have  been  done 
to  measure  the  effectiveness  of  an 
integrated  virtual  reality  system  over  a 
conventional  display  system.  Part  of  the 
reason  may  be  that  the  initial  applications 
for  virtual  reality  have  been 
entertainment  such  as  W  Industries 
Virtuality  video  game  system.  Universities 
and  R&D  centers  such  as  NRaD  who  have 
an  interest  in  the  human  factors  aspects 
may  not  be  able  to  afford  the  resources  in 
time  and  money  needed  to  put  together  a 
virtual  reality  system  for  testing  purposes. 
The  ATAD  project  aims  to  show  that  virtual 
reality  is  useful  for  other  applications 
besides  entertainment  and  can  in  fad  be 
applied  to  the  problems  of  today’s  Navy  by 
helping  to  solve  some  of  the  information 
overload  in  the  ASW  environment. 


